Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add detailed counters to track database perf #2010

Closed

Conversation

loriopatrick
Copy link

@loriopatrick loriopatrick commented May 17, 2024

Exposes prometheus scrape endpoint at /v0/metrics. Runtime overhead is very minimal as we're simply updating a few atomic counters on each call.

Motivation

Currently difficult to see if performance blips in production are due to database slow downs, lock contentions, thread pool queue, or other parts of the code. Having detailed counters allows for easy investigations.

Change Summary

Add metrics module in addon. Metric counters use Atomic numbers and should have very minimal runtime overhead.

Merge Checklist

Choose all relevant options below by adding an x now or at any time before submitting for review

Additional Context

If this is a relatively large or complex change, provide more details here that will help reviewers


PR-Codex overview

This PR updates the codebase to introduce a new metrics module, adds Counters functionality for tracking store actions, and refactors store definitions.

Detailed summary

  • Added metrics module and Counters functionality for tracking store actions.
  • Refactored store definitions to include debug names.
  • Updated references to statsd with metrics.
  • Added rsCountersString function.
  • Implemented store action tracking and printing functionality.

The following files were skipped due to too many changes: apps/hubble/src/addon/src/metrics/counters.rs, apps/hubble/src/addon/src/store/store.rs

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

Copy link

changeset-bot bot commented May 17, 2024

⚠️ No Changeset found

Latest commit: f350876

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link

vercel bot commented May 17, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
hub-monorepo ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 17, 2024 4:50pm

@@ -938,6 +983,20 @@ impl Store {
cached_count: u64,
units: u64,
) -> Result<Vec<HubEvent>, HubError> {
// TODO: uncomment this code and monitor in production
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't want to combine multiple changes in same commit / PR. After merge and monitoring is in place I recommend uncommenting this. If this PR is not going to be merged I recommend making this change as benchmarking showed it massively improves performance. There might be an issue with large prune jobs blocking merges so increasing FID_LOCKS_COUNT might be good to ensure there isn't an unnecessary lock conflict.

Exposes prometheus scrape endpoint at /v0/metrics. Runtime overhead
is very minimal as we're simply updating a few atomic counters on each
call.
@sds
Copy link
Member

sds commented Sep 26, 2024

Closing due to age, and not actively being pursued. Thanks for the work here! We'll pick this back up at some point.

@sds sds closed this Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants